IDE l1a: process multiple l0 files to produce one 10day CDF. #3178
Open
lacoak21 wants to merge 11 commits into
Open
IDE l1a: process multiple l0 files to produce one 10day CDF. #3178lacoak21 wants to merge 11 commits into
lacoak21 wants to merge 11 commits into
Conversation
Contributor
There was a problem hiding this comment.
Pull request overview
This PR updates the IDEX processing pipeline to use 10-day window descriptors (replacing -1week) and adds multi-file L0 ingestion for L1A so multiple packet files can be concatenated and filtered into a single 10-day output window.
Changes:
- Renames IDEX product descriptors from
*-1weekto*-10daysacross processing, CDF global attrs config, and tests. - Adds
idex_l1a(packet_files, start_date)to parse multiple L0 files, concatenate per product type, and filter epochs to the configured 10-day window. - Introduces an IDEX-provided 10-day window lookup CSV and helper
get_10_day_window_end_date()to determine window end dates.
Reviewed changes
Copilot reviewed 12 out of 13 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| imap_processing/idex/idex_l1a.py | Adds multi-file L1A entrypoint and window filtering; changes PacketParser.data to a dict keyed by product. |
| imap_processing/idex/idex_utils.py | Adds get_10_day_window_end_date() that reads the IDEX 10-day window lookup CSV. |
| imap_processing/idex/idex_constants.py | Adds IDEX_10_DAY_RANGES_PATH pointing at the new CSV. |
| imap_processing/idex/idex_10_day_CDF_names.csv | New lookup table mapping start_date -> end_date for IDEX 10-day windows. |
| imap_processing/cli.py | Switches IDEX L1A to call idex_l1a() and updates descriptors used in dependency queries. |
| imap_processing/cdf/config/imap_idex_global_cdf_attrs.yaml | Updates IDEX Logical_source/Data_type strings to use -10days naming. |
| imap_processing/idex/idex_l1b.py | Updates descriptor handling to sci-10days / msg-10days and adjusts global attrs for MSG. |
| imap_processing/idex/idex_l2b.py | Updates only the example snippet (docstring) to reference sci-10days. |
| imap_processing/tests/idex/conftest.py | Updates fixtures to use new PacketParser.data[...] keys and -10days filenames. |
| imap_processing/tests/idex/test_idex_l1a.py | Updates expected filenames and adds unit tests for window end-date + date conversion. |
| imap_processing/tests/idex/test_idex_l1b.py | Updates expected Logical_source/filenames and MSG descriptor usage. |
| imap_processing/tests/idex/test_idex_l2a.py | Updates expected Logical_source/filenames and science descriptor usage. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
tech3371
reviewed
May 11, 2026
tech3371
approved these changes
May 12, 2026
Contributor
tech3371
left a comment
tech3371
left a comment
There was a problem hiding this comment.
code changes looks good. I didn't look closely at tests.
tech3371
reviewed
May 12, 2026
lacoak21
force-pushed
the
idex_l1a_process_multiple_l0_files_filter_and_sort
branch
2 times, most recently
from
4680cab to
00a71f2
Compare
lacoak21
force-pushed
the
idex_l1a_process_multiple_l0_files_filter_and_sort
branch
from
00a71f2 to
ff50c4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Change Summary
closes #3174
closes #IMAP-Science-Operations-Center/sds-data-manager#1303
Overview
Decom each l0 packet file, concat all of them together for each IDEX product type, and filter out events that are not in the 10-day window.
File changes
Testing
TODO